Methods and systems for detecting photograph replacement in a photo identity document

ABSTRACT

Disclosed are computer-implemented methods, non-transitory computer-readable media, and systems for detecting identity document photograph replacement. One computer-implemented method includes receiving image data of a cropped photograph region of an identity document (ID), where the ID comprises a photograph, and where the cropped photograph region comprises at least the photograph. Using a multiclass classification model and as a predicted ID category, an ID category is predicted based on the image data of the cropped photograph region, where the predicted ID category corresponds to a predefined class of a set of predefined classes in the multiclass classification model. The predicted ID category is compared with an indicated category of the ID. In response to a calculated mismatch between the predicted ID category and the indicated category of the ID, a determination is made that the cropped photograph region includes a replaced photograph.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Singapore Patent Application No.10202006694P, filed on Jul. 14, 2020, which is hereby incorporated byreference in its entirety.

TECHNICAL FIELD

The present specification relates broadly, but not exclusively, tomethods and systems for detecting photograph replacement in a photoidentity document.

BACKGROUND

Electronic-Know Your Customer (eKYC) is a digital due diligence processperformed electronically by a business, which can verify theauthenticity of its clients for assessing potential risks of illegalintentions towards the business relationship. To complete an eKYCprocess, a person may need to submit an image of a government-recognizedphoto identity document (ID) to prove his/her identity. In this context,identity fraud can take place when a fraudster uses another person'spersonal information, without authorization, to defraud or commit othercrimes.

One of the most common ID frauds for photo IDs is to replace thephotograph of the ID owner with a photograph of a different person. Todate, Artificial Intelligence (AI) techniques have been used to detectsuch type of ID fraud. A conventional approach to detect photographreplacement in IDs is to treat it as a binary classification task, bytraining a binary classification model and outputting two classes: realID and fake ID with photograph replacement. However, data labelling workinvolved in training the model can be quite time consuming. Given an IDthat is in a card form, at least four steps are required to produce onesample of a fake ID card: getting an ID card, printing a photograph of adifferent person, placing the printed photograph on the ID card to coverthe original photograph, and capturing an image of the resultant fake IDcard and saving it for training. To obtain a model of high accuracy, atleast thousands of samples are required, which can be highly timeconsuming and unlikely to be completed in a short period of time.

SUMMARY

Described embodiments provide methods, apparatuses, and systems fordetecting photograph replacement in a photo identity document (ID). Insome embodiments, the method can train a classification model ofmultiple classes based on samples of cropped photograph regions of IDs,each class corresponding to the category of the ID (e.g., a UnitedStates driver license, a Singapore National registration identity card,a mainland Chinese passport, or the like). A sample can be obtained bycropping the photograph region from an image of an ID and saving it as anew image. In some embodiments, a cropped photograph region can alsoinclude some background surrounding the photograph to comprehensivelydepict the style and features of the photograph region of the specificcategory of ID. In some embodiments, the classification model can betrained with convolutional neural network (CNN).

In some implementations, when an image of an ID is received, thephotograph region can be cropped from the image and used as an input tothe trained multiclass classification model. In some implementations,the photograph region may be obtained by pre-processing the image of theID to generate an aligned image of the ID first, then cropping thephotograph region from the aligned image based on the ID category andthe photograph location information corresponding to the ID category.The model then predicts the ID category based on the cropped photographregion. If a real ID is used, the predicted category of the ID shouldmatch the category of the ID as indicated. Instead, if a fake ID with areplaced photograph is used, the predicted category of the ID should bedifferent from the indicated category of the ID, because the croppedphotograph region (which includes a replaced photograph) looks differentfrom the cropped photograph region of that category of ID as indicated.

According to one embodiment, there is provided a method for detectingphotograph replacement in a photo ID. The method includes: receivingimage data of a cropped photograph region of an ID, wherein the IDcomprises a photograph and the cropped photograph region comprises atleast the photograph; predicting, by a multiclass classification model,an ID category based on the image data of the cropped photograph region,wherein the predicted ID category corresponds to one of a set ofpredefined classes in the multiclass classification model; comparing thepredicted ID category with an indicated category of the ID; and inresponse to the predicted ID category mismatching the indicated categoryof the ID, determining that the cropped photograph region includes areplaced photograph.

In some implementations, the indicated category of the ID can be enteredby the user during an eKYC process or selected from a list of IDcategories. Possible ID categories can include national ID cards,passports, driver licenses from one or more countries. Correspondingly,the multiclass classification model may have a set of predefined classesthat represent the different ID categories. In implementations, when acropped photograph region of a new ID is received, the model may outputa score for each class, and the class with a highest score is determinedas the ID category based on the cropped photograph region. In someimplementations, a predetermined threshold (e.g., 95 out of 100) can beset for the highest score, such that a conclusion of no photographreplacement is made if the highest score reaches the threshold.

According to other embodiments, one or more of these general andspecific embodiments may be implemented using an apparatus including aplurality of modules, a system, a method, or a computer-readable media,or any combination of devices, systems, methods, and computer-readablemedia. The foregoing and other described embodiments can each,optionally, include some, none or all of the following embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments and implementations are provided by way of example only, andwill be better understood and readily apparent to one of ordinary skillin the art from the following written description, read in conjunctionwith the drawings, in which:

FIG. 1 is a flow chart illustrating an example of a method for detectingphotograph replacement in a photo identity document (ID), according toan embodiment.

FIG. 2 is a flow diagram illustrating an example of an implementation ofthe method in FIG. 1, according to an embodiment.

FIG. 3 is a schematic of an example of ID card alignment, according toan embodiment.

FIG. 4 is a schematic diagram of an example of modules of an apparatusfor detecting photograph replacement in a photo ID, according to anembodiment.

FIG. 5 is a block diagram of an example of a computer system suitablefor executing at least some steps of the example methods shown in FIGS.1 and 2, according to an embodiment.

Skilled artisans will appreciate that elements in the figures areillustrated for simplicity and clarity and have not necessarily beendepicted to scale. For example, the dimensions of some of the elementsin the illustrations, block diagrams or flowcharts may be exaggerated inrespect to other elements to help to improve understanding of thepresent embodiments.

DETAILED DESCRIPTION

Embodiments will be described, by way of example only, with reference tothe drawings. Like reference numerals and characters in the drawingsrefer to like elements or equivalents.

Some portions of the description which follows are explicitly orimplicitly presented in terms of algorithms and functional or symbolicrepresentations of operations on data within a computer memory. Thesealgorithmic descriptions and functional or symbolic representations arethe means used by those skilled in the data processing arts to conveymost effectively the substance of their work to others skilled in theart. An algorithm is here, and generally, conceived to be aself-consistent sequence of steps leading to a desired result. The stepsare those requiring physical manipulations of physical quantities, suchas electrical, magnetic or optical signals capable of being stored,transferred, combined, compared, and otherwise manipulated.

Unless specifically stated otherwise, and as apparent from thefollowing, it will be appreciated that throughout the presentspecification, discussions utilizing terms such as “receiving”,“obtaining”, “determining”, “predicting”, “calculating”, “training”,“matching”, “generating”, “detecting”, “classifying”, “setting”,“defining”, “comparing”, “processing”, “updating”, “entering”,“selecting”, “authenticating”, “providing”, “inputting”, “outputting”,or the like, refer to the action and processes of a computer system, orsimilar electronic device, that manipulates and transforms datarepresented as physical quantities within the computer system into otherdata similarly represented as physical quantities within the computersystem or other information storage, transmission or display devices.

The present specification also discloses apparatuses for performing theoperations of the methods. Such apparatuses may be specially constructedfor the required purposes, or may comprise a computer or other deviceselectively activated or reconfigured by a computer program stored inthe computer. The algorithms and displays presented herein are notinherently related to any particular computer or other apparatus.Various machines may be used with programs in accordance with theteachings herein. Alternatively, the construction of more specializedapparatus to perform the required method steps may be appropriate. Thestructure of a computer suitable for executing the variousmethods/processes described herein will appear from the descriptionbelow.

In addition, the present specification also implicitly discloses acomputer program, in that it would be apparent to the person skilled inthe art that the individual steps of the method described herein may beput into effect by computer code. The computer program is not intendedto be limited to any particular programming language and implementationthereof. It will be appreciated that a variety of programming languagesand coding thereof may be used to implement the teachings of thespecification contained herein. Moreover, the computer program is notintended to be limited to any particular control flow. There are manyother variants of the computer program, which can use different controlflows without departing from the scope of the specification.

Furthermore, one or more of the steps of the computer program may beperformed in parallel rather than sequentially. Such a computer programmay be stored on any computer readable medium. The computer readablemedium may include storage devices such as magnetic or optical disks,memory chips, or other storage devices suitable for interfacing with acomputer. The computer readable medium may also include a hard-wiredmedium such as exemplified in the Internet system, or wireless mediumsuch as exemplified in the GSM mobile telephone system. The computerprogram when loaded and executed on such a computer effectively resultsin an apparatus that implements the steps of the preferred method.

The present specification may also be implemented as hardware modules.More particularly, in the hardware sense, a module is a functionalhardware unit designed for use with other components or modules. Forexample, a module may be implemented using discrete electroniccomponents, or it can form a portion of an entire electronic circuitsuch as an Application Specific Integrated Circuit (ASIC) or FieldProgrammable Gate Array (FPGA). Numerous other possibilities exist.Those skilled in the art will appreciate that the system can also beimplemented as a combination of hardware and software modules.

Photograph replacement detection in the act or process of verifying theauthenticity of an ID can be considered as a form of fraud detection orfake identity detection, in which legitimacy of users are verified andpotential fraudsters may be detected before fraudulent acts are carriedout. Effective identity authentication can enhance data security ofsystems by permitting only authenticated users to access its protectedresources. Embodiments seek to provide methods and systems for detectingphotograph replacement in photo IDs thereby detecting fake ID imagesuploaded by fraudsters. Advantageously, financial risks such as moneylaundering and fraud can be effectively reduced or eliminated.

The techniques described in this specification produce one or moretechnical effects. A method is provided to detect photograph replacementof a photo ID by utilizing a multiclass classification model. Inimplementations, the multiclass classification model may include a setof predefined classes corresponding to various categories of IDs.Different from a binary classification technique, which classifies IDsas real ID or fake ID (i.e., ID with a replaced photograph) and involvesintensive data labelling work, no work of data labeling of real/fake IDis required for training the multiclass classification model. Further,the multiclass classification model does not require samples of fake IDsfor training, which advantageously eliminates the effort of producingnumerous fake ID samples in order to achieve a high accuracy.

FIG. 1 is a flow chart 100 illustrating an example method for detectingphotograph replacement in a photo ID including the following steps:

-   -   110: receiving image data of a cropped photograph region of an        ID, wherein the ID includes a photograph and the cropped        photograph region includes at least the photograph;    -   120: predicting, by a multiclass classification model, an ID        category based on the image data of the cropped photograph        region, wherein the predicted ID category corresponds to one of        a set of predefined classes in the multiclass classification        model;    -   130: comparing the predicted ID category with an indicated        category of the ID;    -   135: determining if the predicted ID category matches the        indicated category of the ID; and    -   150: in response to the predicted ID category mismatching the        indicated category of the ID, determining that the cropped        photograph region includes a replaced photograph.

At step 110, image data of a cropped photograph region of a photo ID isreceived. A photo ID can be in a form of a card (such as a nationalidentity cards or a driver license), a document (such as a passport or abirth certificate), or the like. The cropped photograph region includesat least the photograph on the photo ID. In some implementations, areaof the cropped photograph region may be larger than the size of thephotograph, thereby additionally including some background surroundingthe photograph. Advantageously, some features of the particular categoryof ID can be included in the cropped photograph region, which can welldescribe a style of the photograph region and provide useful informationduring classification. Such features can include colors and patterns inthe photograph region, security features, background color of thephotograph, or the like. Depending on the category of the ID, some IDsmay also have texts surrounding the photograph, such as ID owner'sprinted name, gender, date of birth, country of birth, country of issue,or ID number. Some IDs may have images around the photograph, such as IDowner's signature, national emblem of the country of issue, or a smallerversion of the photograph. As features and styles of an ID are uniqueand difficult to falsify, including these features in the croppedphotograph region can contribute to effective detection of photographreplacement on the ID.

To include a background surrounding the photograph, in someimplementations, assuming the photograph has a height of H and a widthof W, the height of the cropped photograph region may be determined asbetween H and R_(H)*H and width may be determined as between W andR_(W)*W (R_(H), R_(W)>1). For example, the cropped photograph region mayhave a height of 1.2*H and a width of 1.2*W. One may appreciate thatR_(H) and R_(W) are predetermined ratios that can be same or different,as long as the photograph is included therein. The implementations arenot limited.

At step 120, based on the image data of the cropped photograph region,an ID category is predicted by the multiclass classification model. Themodel can have a set of predefined classes, with each classcorresponding to a specific category of ID (e.g. Class 1—a SingaporeNational Registration Identity Card, Class 15—a Hong Kong PermanentIdentity Card, Class 30—a US driver license, Class 55—a PhilippinesNational ID card, Class 71—an Australian passport, and so on). Thepredicted ID category will be one of the predefined classes, based onthe result of the classification. In some implementations, predictingthe ID category based on the image data of the cropped photograph regionincludes: inputting the image data to the multiclass classificationmodel, and outputting a score for each class in the set of thepredefined classes. The respective scores may reflect probabilities ofthe cropped photograph region from a respective ID category. Thereafter,a class with a highest score can be determined as the predicted IDcategory. One may appreciate that this is one of the many possibleimplementations for predicting the ID category and is non-limiting.

At step 130, the predicted ID category is compared with an indicatedcategory of the ID. In some implementations, the indicated category ofthe ID can be entered or selected by a user during an identityverification process (such as eKYC). Alternatively, the identityverification system may indicate to the user to upload a certaincategory of the ID, which will be the indicated category of the ID. Acomparison of the predicted ID category outputted by the model and theindicated category of the ID is made at step 135. If the predicted IDcategory mismatches the indicated category of the ID, the methodproceeds to step 150 and determines that the cropped photograph regionincludes a replaced photograph. If the predicted ID category matches theindicated category of the ID, the method may proceed to step 160 anddetermine that the photograph in the cropped photograph region is anauthentic photograph on the photo ID.

Optionally, after determining that the predicted ID category matches theindicated category of the ID, the method may include additional steps140 and 145 before concluding if the photograph in the croppedphotograph region has been replaced. In some implementations, theadditional steps involve predetermining a threshold value in relation tothe outputted scores and comparing the highest score with thepredetermined threshold. If the highest score is below the predeterminedthreshold, the method proceeds to step 150 and determines that thecropped photograph region includes a replaced photograph. If the highestscore is equal to or above the predetermined threshold, the method mayproceed to step 160 and determine that the photograph in the croppedphotograph region is an authentic photograph on the photo ID. Theseadditional steps 140 and 145 are advantageous in situations when areplaced photograph has been used and the multiclass classificationmodel outputs low scores for all the classes (e.g., ≤40 out of 100 forall scores), which implies that the probability of the croppedphotograph region from any category of ID is low. In such situations, anID category will still be predicted at step 120 by selecting the highestscore (e.g. 40), and there is a chance the predicted ID category maymatch with the indicated category of the ID. With the additional steps,in response to the predicted ID category matching the indicated categoryof the ID, if the highest score (e.g. 40) is below the preset threshold(e.g. 95 out of 100), the method will still proceed to step 150 anddetermine that the cropped photograph region includes a replacedphotograph. One may appreciate that false positives in the photographreplacement detection can be reduced by setting and tuning thethreshold.

The present specification may further provide methods, apparatuses, andsystems for training the multiclass classification model based onsamples of cropped photograph regions of IDs. Assuming samples ofcropped photograph regions of N categories of photo IDs are provided, aclassification model of N classes can be trained with each classcorresponding to each category of the photo IDs. In some embodiments,the classification model can be trained with convolutional neuralnetwork (CNN), which is one of the deep neural networks widely used toanalyze visual imagery. One may appreciate that the multiclassclassification model can be efficiently trained and developed withoutany data labelling required in a binary classification task. Further,more image data of cropped photograph regions can be collected when theclassification model is implemented to detect photograph replacement ofreceived IDs. Optionally, the samples of cropped photograph regions usedfor training the classification model may be updated by including newcropped photograph regions of the received IDs, which may enhance thefuture training models and improve the accuracy.

FIG. 2 is an example flow diagram 200 illustrating an implementation ofthe method in FIG. 1. The left part of the flow diagram directs totraining the multiclass classification model. In the beginning of theprocess, there is provided an image set of N categories of ID cards. Anexample is shown for an image 210 of a Hong Kong Permanent ID card. Acropped photograph region 215 can be obtained by cropping from the image210. One may appreciate that the cropped photograph region 215 includesthe photograph as well as some background surrounding the photograph. Ina similar manner, samples of N types cropped photograph regions can beobtained from the image set of the N categories of photo IDs. Next, aclassification model is trained to classify N types of photographregions.

The trained model can be used for detecting photograph replacement of areceived ID. A cropped photograph region 225 is obtained from an image220 of the received ID. In this case, the received ID is a fake HongKong ID which includes a replaced photograph from a Philippines UnifiedMulti-Purpose ID card. By inputting the cropped photograph region 225into the classification model, the model will likely predict an IDcategory to be Philippines Unified Multi-Purpose ID, which is differentfrom the indicated category of Hong Kong ID. As such, it is determinedthat the received ID is a fake ID with a replaced photograph.

In implementations, the image 220 of the received ID may be uploaded viawebpages or mobile applications. The cropped photograph region 225 canbe obtained by cropping a photograph region 225 from the image 220 ofthe ID based on the indicated category of the ID and photograph locationinformation corresponding to the indicated category. For example, if thereceived ID is indicated to be a Hong Kong ID card, and on Hong Kong IDcard the photo region may be at (0.10*h→0.75*h, 0.05*w→0.35*w) where(h,w) is (height, width) of ID card, then one can use this photographlocation information to crop the photograph region from an image of aHong Kong ID card.

In some implementations, obtaining the cropped photograph region canfurther include pre-processing the image of the ID to generate analigned image of the ID. FIG. 3 is a schematic 300 of an example of IDcard alignment. As shown in the figure, an image 302 is received whichincludes an ID card 310, and the ID card 310 is tilted in the image. Inthe next step, the four corner points of the ID card 310 can be detectedby machine learning techniques, such as a CNN regression model. Byperforming alignment, an aligned ID card image 310A can be generated.One can then obtain the cropped photograph region using theaforementioned method, by cropping a region based on the photographlocation information of the specific category of ID. The techniques forcropping a photograph region from a photo ID can be used for both thereceived ID and the image set of IDs for training the model.

One may appreciate that training the model and using the model forphotograph replacement detection can be two separate processes,performed by either the same party or different parties. Further, thephotograph replacement detection method can be implemented alone or incombination with other methods of identity verification and identityproofing. The implementations are not limited.

FIG. 4 is a schematic diagram of an example apparatus 400 includingmodules for detecting photograph replacement in a photo ID. Theapparatus 400 at least includes a receiving module 410, a classificationmodule 420, a comparison module 430, and a determining module 440. Withreference to FIG. 1 and FIG. 2, the receiving module 410 can beconfigured to receive image data of a cropped photograph region of anID, and receive information on an indicated category of the IDentered/selected by a user. The classification module 420 can beconfigured to predict an ID category based on the image data of thecropped photograph region received by the receiving module 410. Thecomparison module 430 can be configured to compare the predicted IDcategory with the indicated category of the ID, and compare the scorecorresponding to the predicted ID category with a predeterminedthreshold. The determining module 440 can be configured to determine ifthe cropped photograph region includes a replaced photograph based on ifthe predicted ID category matches the indicated category of the ID, andbased on if the score is above the predetermined threshold. Theapparatus 400 may additionally include a training module configured totrain the multiclass classification model. The apparatus 400 may alsoinclude an ID image processing module configured to pre-process areceived ID image, generate an aligned ID image, and crop a photographregion. The apparatus 400 may also include an output module configuredto output the photograph replacement detection result obtained by thedetermining module 440. One or more or any combination of these modulescan be part of an apparatus for detecting photograph replacement in aphoto ID.

The system, apparatus, module, or unit illustrated in the previousembodiments can be implemented by using a computer chip or an entity, orcan be implemented by using a product having a certain function. Atypical embodiment device is a computer (and the computer can be apersonal computer), a laptop computer, a cellular phone, a camera phone,a smartphone, a personal digital assistant, a media player, a navigationdevice, an email receiving and sending device, a game console, a tabletcomputer, a wearable device, or any combination of these devices. Themodules described as separate parts may or may not be physicallyseparate, and parts displayed as modules may or may not be physicalmodules, may be located in one position, or may be distributed on anumber of network modules. Some or all of the modules can be selectedbased on actual demands to achieve the objectives of the solutions ofthe specification. A person of ordinary skill in the art can understandand implement the embodiments of the present application withoutcreative efforts.

FIG. 5 is a block diagram of an example computer system 500 suitable forexecuting at least some steps of the example methods shown in FIGS. 1and 2. The following description of the computer system/computing device500 is provided by way of example only and is not intended to belimiting.

As shown in FIG. 5, the example computing device 500 includes aprocessor 502 for executing software routines. Although a singleprocessor is shown for the sake of clarity, the computing device 500 mayalso include a multi-processor system. The processor 502 is connected toa communication infrastructure 506 for communication with othercomponents of the computing device 500. The communication infrastructure506 may include, for example, a communications bus, cross-bar, ornetwork.

The computing device 500 further includes a main memory 504, such as arandom access memory (RAM), and a secondary memory 510. The secondarymemory 510 may include, for example, a storage drive 512, which may be ahard disk drive, a solid state drive or a hybrid drive and/or aremovable storage drive 514, which may include a magnetic tape drive, anoptical disk drive, a solid state storage drive (such as a USB flashdrive, a flash memory device, a solid state drive or a memory card), orthe like. The removable storage drive 514 reads from and/or writes to aremovable storage medium 518 in a well-known manner. The removablestorage medium 518 may include magnetic tape, optical disk, non-volatilememory storage medium, or the like, which is read by and written to byremovable storage drive 514. As will be appreciated by persons skilledin the relevant art(s), the removable storage medium 518 includes acomputer readable storage medium having stored therein computerexecutable program code instructions and/or data.

In an alternative implementation, the secondary memory 510 mayadditionally or alternatively include other similar means for allowingcomputer programs or other instructions to be loaded into the computingdevice 500. Such means can include, for example, a removable storageunit 522 and an interface 520. Examples of a removable storage unit 522and interface 520 include a program cartridge and cartridge interface(such as that found in video game console devices), a removable memorychip (such as an EPROM or PROM) and associated socket, a removable solidstate storage drive (such as a USB flash drive, a flash memory device, asolid state drive or a memory card), and other removable storage units522 and interfaces 520 which allow software and data to be transferredfrom the removable storage unit 522 to the computer system 500.

The computing device 500 also includes at least one communicationinterface 524. The communication interface 524 allows software and datato be transferred between computing device 500 and external devices viaa communication path 526. In various embodiments of the specification,the communication interface 524 permits data to be transferred betweenthe computing device 500 and a data communication network, such as apublic data or private data communication network. The communicationinterface 524 may be used to exchange data between different computingdevices 500 which such computing devices 500 form part an interconnectedcomputer network. Examples of a communication interface 524 can includea modem, a network interface (such as an Ethernet card), a communicationport (such as a serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB),an antenna with associated circuitry and the like. The communicationinterface 524 may be wired or may be wireless. Software and datatransferred via the communication interface 524 are in the form ofsignals which can be electronic, electromagnetic, optical or othersignals capable of being received by communication interface 524. Thesesignals are provided to the communication interface via thecommunication path 526.

As shown in FIG. 5, the computing device 500 further includes a displayinterface 528 which performs operations for rendering images to anassociated display 530 and an audio interface 532 for performingoperations for playing audio content via associated speaker(s) 534.

As used herein, the term “computer program product” may refer, in part,to removable storage medium 518, removable storage unit 522, a hard diskinstalled in storage drive 512, or a carrier wave carrying software overcommunication path 526 (wireless link or cable) to communicationinterface 524. Computer readable storage media refers to anynon-transitory, non-volatile tangible storage medium that providesrecorded instructions and/or data to the computing device 500 forexecution and/or processing. Examples of such storage media includemagnetic tape, CD-ROM, DVD, Blu-Ray™ Disc, a hard disk drive, a ROM orintegrated circuit, a solid state storage drive (such as a USB flashdrive, a flash memory device, a solid state drive or a memory card), ahybrid drive, a magneto-optical disk, or a computer readable card suchas a PCMCIA card and the like, whether or not such devices are internalor external of the computing device 500. Examples of transitory ornon-tangible computer readable transmission media that may alsoparticipate in the provision of software, application programs,instructions and/or data to the computing device 500 include radio orinfra-red transmission channels as well as a network connection toanother computer or networked device, and the Internet or Intranetsincluding e-mail transmissions and information recorded on Websites andthe like.

The computer programs (also called computer program code) are stored inmain memory 504 and/or secondary memory 510. Computer programs can alsobe received via the communication interface 524. Such computer programs,when executed, enable the computing device 500 to perform one or morefeatures of embodiments discussed herein. In various embodiments, thecomputer programs, when executed, enable the processor 607 to performfeatures of the above-described embodiments. Accordingly, such computerprograms represent controllers of the computer system 500.

Software may be stored in a computer program product and loaded into thecomputing device 500 using the removable storage drive 514, the storagedrive 512, or the interface 520. The computer program product may be anon-transitory computer readable medium. Alternatively, the computerprogram product may be downloaded to the computer system 500 over thecommunication path 526. The software, when executed by the processor502, causes the computing device 500 to perform the necessary operationsto execute the method as shown in FIGS. 1 and 2.

It is to be understood that the embodiment of FIG. 5 is presented merelyby way of example to explain the operation and structure of the system500. Therefore, in some embodiments one or more features of thecomputing device 500 may be omitted. Also, in some embodiments, one ormore features of the computing device 500 may be combined together.Additionally, in some embodiments, one or more features of the computingdevice 500 may be split into one or more component parts.

It will be appreciated that the elements illustrated in FIG. 5 functionto provide means for performing the various functions and operations ofthe system as described in the above embodiments.

It will be appreciated by a person skilled in the art that numerousvariations and/or modifications may be made to the present specificationas shown in the specific embodiments without departing from the scope ofthe specification as broadly described. The present embodiments are,therefore, to be considered in all respects to be illustrative and notrestrictive.

What is claimed is:
 1. A computer-implemented method for detectingidentity document photograph replacement, comprising: receiving imagedata of a cropped photograph region of an identity document (ID),wherein the ID comprises a photograph, and wherein the croppedphotograph region comprises at least the photograph; predicting, by amulticlass classification model and as a predicted ID category, an IDcategory based on the image data of the cropped photograph region,wherein the predicted ID category corresponds to a predefined class of aset of predefined classes in the multiclass classification model;comparing the predicted ID category with an indicated category of theID; and in response to a calculated mismatch between the predicted IDcategory and the indicated category of the ID, determining that thecropped photograph region includes a replaced photograph.
 2. Thecomputer-implemented method of claim 1, wherein the cropped photographregion further comprises a background surrounding the photograph.
 3. Thecomputer-implemented method of claim 2, wherein the photograph has aheight of H and a width of W, and the cropped photograph region has aheight between H and R_(H)*H and a width between W and R_(W)*W, whereinR_(H) and R_(W) are predetermined ratios that are greater than
 1. 4. Thecomputer-implemented method of claim 1, wherein receiving image data ofa cropped photograph region of an ID comprises: receiving an image ofthe ID; pre-processing the image of the ID to generate an aligned imageof the ID; and cropping a photograph region from the aligned image ofthe ID based on the indicated category of the ID and photograph locationinformation corresponding to the indicated category of the ID.
 5. Thecomputer-implemented method of claim 1, wherein predicting the IDcategory based on the image data of the cropped photograph regioncomprises: inputting the image data to the multiclass classificationmodel; outputting a score for each predefined class of the set ofpredefined classes; and determining the ID category, wherein the IDcategory corresponds to a predefined class of the set of predefinedclasses with a highest score.
 6. The computer-implemented method ofclaim 5, further comprising: setting a threshold for the highest score;and in response to the predicted ID category matching the indicatedcategory of the ID and the highest score less than the threshold,determining that the cropped photograph region comprises a replacedphotograph.
 7. The computer-implemented method of claim 1, wherein theindicated category of the ID is entered or selected by a user.
 8. Thecomputer-implemented method of claim 1, wherein the set of predefinedclasses comprises national ID cards, passports, driver licenses from oneor more countries.
 9. The computer-implemented method of claim 1,wherein the multiclass classification model is trained by: providingsample images of N categories of photo IDs; generating croppedphotograph regions based on the sample images of N categories of photoIDs; and training a classification model, wherein the classificationmodel has N classes based on the cropped photograph regions, and whereinthe N classes correspond to N categories of photo IDs.
 10. Thecomputer-implemented method of claim 9, wherein generating croppedphotograph regions based on the sample images of N categories of photoIDs comprises: pre-processing at least one sample image of the sampleimages of N categories of photo IDs to generate an aligned image of thephoto ID; and cropping, based on a category of the photo ID andphotograph location information corresponding to the category of thephoto ID, a photograph region from the aligned image of the photo ID.11. The computer-implemented method of claim 9, further comprising:updating the cropped photograph regions from the sample images of Ncategories of photo IDs that were used to train the classification modelby including the image data of the cropped photograph regions.
 12. Thecomputer-implemented method of claim 9, wherein training theclassification model based on the cropped photograph regions is based ona convolutional neural network (CNN) algorithm.
 13. A non-transitorycomputer-readable medium storing one or more instructions executable bya computer system to perform operations for detecting identity documentphotograph replacement, comprising: receiving image data of a croppedphotograph region of an identity document (ID), wherein the ID comprisesa photograph, and wherein the cropped photograph region comprises atleast the photograph; predicting, by a multiclass classification modeland as a predicted ID category, an ID category based on the image dataof the cropped photograph region, wherein the predicted ID categorycorresponds to a predefined class of a set of predefined classes in themulticlass classification model; comparing the predicted ID categorywith an indicated category of the ID; and in response to a calculatedmismatch between the predicted ID category and the indicated category ofthe ID, determining that the cropped photograph region includes areplaced photograph.
 14. The non-transitory computer-readable medium ofclaim 13, wherein the cropped photograph region further comprises abackground surrounding the photograph.
 15. The non-transitorycomputer-readable medium of claim 14, wherein the photograph has aheight of H and a width of W, and the cropped photograph region has aheight between H and R_(H)*H and a width between Wand R_(W)*W, whereinR_(H) and R_(W) are predetermined ratios that are greater than
 1. 16.The non-transitory computer-readable medium of claim 13, whereinreceiving image data of a cropped photograph region of an ID comprises:receiving an image of the ID; pre-processing the image of the ID togenerate an aligned image of the ID; and cropping a photograph regionfrom the aligned image of the ID based on the indicated category of theID and photograph location information corresponding to the indicatedcategory of the ID.
 17. The non-transitory computer-readable medium ofclaim 13, wherein predicting the ID category based on the image data ofthe cropped photograph region comprises: inputting the image data to themulticlass classification model; outputting a score for each predefinedclass of the set of predefined classes; and determining the ID category,wherein the ID category corresponds to a predefined class of the set ofpredefined classes with a highest score.
 18. The non-transitorycomputer-readable medium of claim 17, further comprising operations for:setting a threshold for the highest score; and in response to thepredicted ID category matching the indicated category of the ID and thehighest score less than the threshold, determining that the croppedphotograph region comprises a replaced photograph.
 19. Thenon-transitory computer-readable medium of claim 13, wherein theindicated category of the ID is entered or selected by a user.
 20. Thenon-transitory computer-readable medium of claim 13, wherein the set ofpredefined classes comprises national ID cards, passports, driverlicenses from one or more countries.
 21. The non-transitorycomputer-readable medium of claim 13, wherein the multiclassclassification model is trained by: providing sample images of Ncategories of photo IDs; generating cropped photograph regions based onthe sample images of N categories of photo IDs; and training aclassification model, wherein the classification model has N classesbased on the cropped photograph regions, and wherein the N classescorrespond to N categories of photo IDs.
 22. The non-transitorycomputer-readable medium of claim 21, wherein generating croppedphotograph regions based on the sample images of N categories of photoIDs comprises: pre-processing at least one sample image of the sampleimages of N categories of photo IDs to generate an aligned image of thephoto ID; and cropping, based on a category of the photo ID andphotograph location information corresponding to the category of thephoto ID, a photograph region from the aligned image of the photo ID.23. The non-transitory computer-readable medium of claim 21, furthercomprising operations for: updating the cropped photograph regions fromthe sample images of N categories of photo IDs that were used to trainthe classification model by including the image data of the croppedphotograph regions.
 24. The non-transitory computer-readable medium ofclaim 21, wherein training the classification model based on the croppedphotograph regions is based on a convolutional neural network (CNN)algorithm.
 25. A computer-implemented system for detecting identitydocument photograph replacement, comprising: one or more computers; andone or more computer memory devices interoperably coupled with the oneor more computers and having tangible, non-transitory, machine-readablemedia storing one or more instructions that, when executed by the one ormore computers, cause the one or more computers to perform one or moreoperations comprising: receiving image data of a cropped photographregion of an identity document (ID), wherein the ID comprises aphotograph, and wherein the cropped photograph region comprises at leastthe photograph; predicting, by a multiclass classification model and asa predicted ID category, an ID category based on the image data of thecropped photograph region, wherein the predicted ID category correspondsto a predefined class of a set of predefined classes in the multiclassclassification model; comparing the predicted ID category with anindicated category of the ID; and in response to a calculated mismatchbetween the predicted ID category and the indicated category of the ID,determining that the cropped photograph region includes a replacedphotograph.